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© Stereoscopic image pickup and display apparatus. 

© Binocular parallax of a captured image is detected, and image pickup cameras are controlled so that their 
optic axes converge at an optimum point to ensure that the parallax of the captured image comes within the 
binocular fusing range of a viewer. Further, an optimum fixating point is calculated that enables the viewer to 
perceive the stereoscopic depth of the object over the widest possible range, and control is performed so that 
the fixating point is reproduced at a surface of a stereoscopic image display or at a designated distance from the 
surface. This suppresses the unnatural feel when the viewer views stereoscopic images. 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention 
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The present invention relates to a stereoscopic image pickup and display apparatus for recording and 
displaying a stereoscopic image of an object. 

2. Description of the Related Art 

An example of a prior art stereoscopic image pickup apparatus is shown in Figure 10 
The apparatus shown comprises two ordinary cameras mounted on two separate universal heads the 
two cameras corresponding to the left and right eyes of a human. The cameras are secured to rotation 
controllers 3 and 4, respectively, so that they are rotatable in a horizontal plane. The spacing between the 
two cameras is adjustable on a camera spacing controller 5. The entire camera construction is supported on 

Lh'™' J?? T WS ° f thiS Pri0f 8rt stereosc °P^ im age Pickup apparatus are given in Figures 11(a) 
and 11(b): (a) shows the apparatus in a parallel shooting position where the optic axes of the cameras are 
set parallel to each other, and (b) illustrates a converging shooting position where the optic axes of the 
cameras are tilted so that they intersect each other. The rotation controllers 3 and 4 are operated to rotate 
the respective cameras and set them in the parallel shooting or the converging shooting position The 
camera spacing is adjustable by varying Wc. In shooting an object in three dimensions, the camera spacing 
Wc and the d.stance dx to the converging point greatly affect the quality of the resulting stereoscopic 
image The camera spacing Wc affects the size of the object to be displayed as well as the amount of the 

P f? h t0 ! T 0duced - The distance d * t° *° converging point has a significant effect on the 
distortion of the recorded image as well as on the amount of the horizontal parallax (which affects the 

soThh? £T ^ , VieWer) " the Pri ° r art ^^opic image pickup apparatus, the camera 
spac ng Wc and the distance dx to the converging point are set by an operator based on his experience so 
that the recorded image can be perceived properly when viewed by a viewer. Here, "perceived properly" 
essentially means bringing the horizontal parallax of the stereoscopic image within the fusing range of the 
viewers eyes when the image is viewed by the viewer. The fusing range will be explained in further detail 
in the preferred embodiments of the present invention. Figure 12 is a schematic diagram of a prior art 
stereoscopic image display apparatus for reproducing a stereoscopic image signal recorded by the 
stereoscopic image pickup apparatus. In the display apparatus, two separate images, one for each eye 
recorded by the stereoscopic image pickup apparatus are reproduced by projectors 9 and 10. are polarized 
through two polarizers 11 oriented perpendicular to each other, and are focused onto a screen 12 When 

iSIT'?^ ^ 3 Vi6Wer W6arin9 3 P3ir ° f P0,arizin 9 9 ,asses 13 ' the ri 9ht-eye image is 
reflected only into the viewer's right eye and the left-eye image only into the viewer's left eye 

Figure 13 shows the screen of the stereoscopic image display apparatus as viewed from above If IL 
and IR represent the left- and right-eye spot images obtained by the stereoscopic image pickup apparatus 
a viewer with an interocular distance of We perceives the object virtually lying at the intersection C1 
between the v.sual axes. Thus, the viewer can perceive the depth of the displayed image by binocular 
s ereoscopic vision^ln another prior art stereoscopic image display method, as shown in Figure 14. images 
delayed on two CRTs with crossed polarizers are combined by a half-silvered mirror without using a 
screen, and the combined image is viewed through glasses with the corresponding polarizers. In any case 
the principle of making the viewer perceive the depth of an image is the same. 

Next, the perception of depth will be described in further detail with reference to Figure 15 This 
d.agram explains how a viewer perceives image depth based on the parallax between his left and right 
y ! S ,«V"! ? am ' A> B> and P re P resent ob i ects (P° in ts). CL is the center of rotation of the viewer's left 
f « : u' S Ce " ,er ° f r ° ,a,i0n 0f the viewer ' s ri 9 nt e * e 21 > and FL and FR are foveae of the viewer's 
left and right eyes, respectively (the fovea is the region in the retina where there is the greatest acuity of 
vision; when a human views an object, an image of the object is focused onto the fovea ). Now suppose that 
the viewer focuses his eyes on the point P. In this case, the left and right eyeballs move so that images of 
the point P are focused on the respective foveae . FL and FR. At the same time, images of the point A 
situated nearer to the eyes than the point P are focused on AL and AR on the retinas of the left and right 
eyes Similarly, images of the point B farther from the point P are focused on BL and BR on the retinas of 
he left and right eyes. At this time, the distance ALBL between the images of the point A and point B 
formed on the retina of one eye is different from the corresponding distance ARBR in the other eye. This 
drfference in d.stance corresponds to the distance between the point A and point B in the direction of depth 
In terms of angle, the binocular parallax iBA between the point B and point A is expressed as 7 BA = y B - 
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>A, the amount of which corresponds to the distance between the point A and point B in the direction of 
depth. A human calculates this difference in his brain to perceive the depthwise position of an object, i.e., 
LAB in Figure 15. As the distance to an object increases, a human's perception of the absolute distance LA, 
LB to the point A, B becomes less sensitive while his perception of the relative distance LAB becomes 
5 more sensitive. 

However, in the prior art construction described above, a viewer perceives the object lying at position 
C1 when the actual images are focused on the display screen 12. That is, the actually displayed images 
and the image as viewed by the viewer are at different fixating positions of the eyes. This gives rise to a 
conflict between the convergence point of the visual axes of the viewer's eyes (both eyes turn inward with 

70 the eyeballs directed to the nearer position C1) and the accomodation point of the eyes (fixating of the 
eyes). If the amount of this conflict becomes too large, a human becomes unable to perceive the left and 
right images as a single image, and perceives them as a double image. Even if the image does not appear 
double, the resulting stereoscopic image may appear very unnatural and may make the viewer feel sick. 
To describe in further detail, in Figure 15 where the viewer focuses his eyes on the point P, a human 

15 can calculate the depth when the amount of the binocular parallax 5 is within a range of ±0.5*; if the 
depthwise distance becomes larger (the absolute value of the binocular parallax 5 becomes larger), the 
sense of depthwise distance decreases, and if the distance increases further, the object will appear double, 
making it difficult to perceive the depth and the image very unpleasant to view. Here, suppose that the 
points A, B, and P in Figure 15 are displayed on the stereoscopic image display apparatus of Figure 14. As 

20 the position of the reproduced stereoscopic image moves farther away from the position 28 of the CRT 
surface where the images are displayed, the displacement between the fixating position (CRT surface) of 
the eye lens and the depthwise position of the presented image becomes larger. Human vision has a 
threshold of tolerance for the displacement between the CRT surface and the stereoscopic image display 
position. If the displacement becomes larger than this threshold, images of the intended object cannot be 

25 focused on the foveae of both eyes, and the viewer perceives them as a double image. 

Considering the above facts, it is clear that a human's depth perception by binocular stereoscopic 
vision is limited to within a certain range centering around the fixating point P. To perceive a broader three- 
dimensional world, a human moves the fixating point P in depthwise direction, perceives three-dimensional 
geometries around respective fixating points, and merges them to perceive the entire perspective. However, 

30 with the prior art stereoscopic image display apparatus, since there are tolerable limits to the displacement 
between the fixating position (CRT surface) of the eye lens and the depthwise position of the presented 
image, it is not possible to observe the entire space around the object. Furthermore, since there always is a 
displacement between the fixating position (CRT surface) of the eye lens and the depthwise position of the 
presented image, it is difficult to move the fixating point quickly. These factors combined, the prior art 

35 stereoscopic image display apparatus has not been able to present images that are easy on the eyes and 
that have good visibility. 

To compensate for these shortcomings, in operating the prior art stereoscopic image pickup apparatus 
the operator had to measure the shooting distance in the field and determine Wc and dx in Figure 11 by 
accounting for the size of the stereoscopic image display apparatus used and the viewing distance from the 

40 audience, thereby avoiding the above-stated situation as far as possible. However, the calculations were not 
accurate since they were based on rough measurement of the shooting distance or on the experience of the 
operator, and camera parameters were set focusing on a single object so that the object could be viewed in 
three dimensions with both eyes without strain. Further, where the depth to be reproduced varies in the time 
domain, such as the object moving in depthwise direction, it was not possible to determine We and dx 

45 optimally. 

SUMMARY OF THE INVENTION 

The present invention provides a stereoscopic image pickup and display apparatus consisting of a 
50 stereoscopic image pickup apparatus and a stereoscopic image display apparatus in combination: the 
stereoscopic image pickup apparatus comprises an image pickup device for capturing an object from 
different viewpoints; a converging angle moving mechanism for varying the converging angle of the image 
pickup device; a signal processor for extracting image data from the image pickup device; a parallax 
calculator for calculating the parallax of an image by using an output of the signal processor; a parallax 
55 processor for detecting the smallest value (the parallax of the nearest object) out of outputs from the 
parallax calculator; a depth display position calculator for calculating from an output of the parallax 
processor the depthwise position of an object to be reproduced at the nearest point when a captured image 
is displayed by the stereoscopic image display apparatus; a fusing range verifier for verifying on the basis 
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of an output of the depth display position calculator whether the nearest point is within a fusional range of a 
viewer when the viewer views the reproduced image; and a converging angle controller for controlling the 
converging angle moving mechanism on the basis of an output of the fusing range verifier so that the object 
at the nearest point is positioned within the fusiona! range of the viewer; and the stereoscopic image display 

5 apparatus comprises a parallax calculator for calculating the binocular parallax or three-dimensional position 
of an object; an fixating point calculator for calculating an average value for the depthwise position of the 
object over an entire screen, or an weighted average value with greater weight given to the center of the 
screen, from an output of the parallax calculator; and a parallax controller for performing control so that the 
depthwise position indicated by an output of the fixating point calculator is reproduced at a surface of a 

io stereoscopic image display or at a designated distance from the surface. 

With the above construction of the present invention, the binocular parallax of a captured image is 
detected, based on which the depthwise position of an object at the nearest point to the viewer is 
calculated, and the converging angle moving mechanism is controlled so that the nearest point comes 
within the fusional range of the viewer viewing the reproduced image; furthermore, from the binocular 

75 parallax of the image, an optimum fixating point is calculated that enables the viewer to perceive the 
stereoscopic depth of the object over the widest possible range, and control is performed so that the 
fixating point is reproduced at the surface of the stereoscopic image display or at a designated distance 
from the surface. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a diagram showing the configuration of a stereoscopic image pickup apparatus according to a 
first embodiment of the invention; 

Figure 2 is a set of diagrams showing an imaging coordinate system and a display coordinate system, 
25 respectively, for a stereoscopic image; 

Figure 3 is a diagram showing allowable ranges of a human's adjusting mechanism and converging 
mechanism; 

Figure 4 is a diagram showing a method of calculating the parallax; 

Figure 5 is a diagram showing a binocular fusing range in a stereoscopic image display apparatus (at 
30 viewing distance > 71 cm); 

Figure 6(a) is a diagram illustrating the definitions of distance dx to the converging point and angle 0 of 
optic axis rotation, and 6(b) is a diagram plotting distance dx to the converging point as a function of 
angle 6 of optic axis rotation; 

Figure 7 is a diagram showing the configuration of a stereoscopic image pickup apparatus according to a 
35 second embodiment of the invention; 

Figure 8 is a diagram showing the configuration of a stereoscopic image pickup apparatus according to a 
third embodiment of the invention; 

Figure 9 is a diagram showing a binocular fusing range in a stereoscopic image display apparatus (at 
viewing distance < 71 cm); 
40 Figure 1 0 is a schematic diagram of a prior art stereoscopic image pickup apparatus; 

Figure 11 is a set of diagrams showing stereoscopic image shooting methods (parallel shooting and 
converging shooting); 

Figure 12 is a schematic diagram of a stereoscopic image display apparatus; 
Figure 13 is a diagram showing a stereoscopic image display position; 
45 Figure 1 4 is an external view of a prior art stereoscopic image display apparatus; 

Figure 15 is a diagram explaining how a viewer perceives depth based on binocular parallax; 

Figure 16 is a diagram showing the configuration of a stereoscopic image display apparatus according to 

a fourth embodiment of the invention; 

Figure 17 is a diagram for explaining a technique for obtaining weighted average of parallax; 
so Figure 18 is a diagram showing the operation of a parallax controller 12; 

Figure 19 is a diagram showing the configuration of a stereoscopic image display apparatus according to 
a fifth embodiment of the invention; 

Figure 20 is a diagram showing the operation of a line-of-vision sensor; 

Figure 21 is a diagram showing a method of calculating the coordinates of an fixating point in a 
55 displayed image; 

Figure 22 is a diagram showing a method of calculating the position of the fixating point; 

Figure 23 is a diagram showing a method of calculating the control amount for horizontal read timing of 

left and right images; 
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Figure 24 is a diagram showing the configuration of a stereoscopic image display apparatus according to 
a sixth embodiment of the invention; and 

Figure 25 is a diagram showing the configuration of a stereoscopic image display apparatus according to 
a seventh embodiment of the invention. 

5 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Figure 1 is a diagram showing the configuration of an image pickup section in a stereoscopic image 
pickup and display apparatus according to a first embodiment of the present invention. In Figure 1, the 

70 numerals 1 and 2 are lenses, 3a and 3b are camera bodies, 4a and 4b are rotation controllers, and 5 is a 
camera spacing controller. These components are the same as those used in the prior art. The numerals 
14a and 14b are signal processors each for converting the camera signal into the luminance signal, 15 is a 
parallax calculator for calculating horizontal parallax from left and right images, 16 is a parallax processor for 
detecting the smallest parallax, 17 is a depth display position calculator for calculating the spatial position of 

75 an object nearest to the viewer, 18 is a fusing range verifier for verifying whether the spatial position of the 
object nearest to the viewer is within the fusing range of the viewer's eyes, and 19 is a converging angle 
controller for controlling the rotation controllers 4a, 4b so that the spatial position of the object nearest to the 
viewer comes within the fusing range of the viewer's eyes. 

Before describing the operation of the thus constructed stereoscopic image pickup device and display 

20 section of the present embodiment, we will first describe the modeling theory of the image pickup system 
and display system and the characteristics of a viewer's binocular fusing range. 

First, models of the image pickup system and display system will be described with reference to Figure 
2. Figure 2(a) shows a model of the image pickup system. The image pickup cameras are arranged 
symmetric to each other with their optic axes directed to a converging point F (0\ dx, 0). In the coordinate 

25 system shown here, the centerpoint of the line joining the two cameras is the origin, the two cameras are 
arranged along the x-axis, the direction of depth is plotted along the y-axis, and the direction of height along 
the 2-axis. The left and right lenses are positioned at (-Wc, 0, 0) and (Wc, 0,- 0), respectively, on the x-axis. 
The focal length of each lens with respect to the imaging surface (the position of the imaging device or film) 
7, 8 is denoted by f. In this arrangement, when dx = », the optic axes of the two cameras are parallel to 

30 each other (shown by dashed lines in Figure 2(a)). Figure 2(b) shows a model of the display system (a 
model of a stereoscopic image display apparatus). In the display coordinates system, the origin represents 
the centerpoint between the two eyes of the viewer, the line joining the centers of the two eyes is taken as 
the x-axis, the direction of depth is plotted along the y-axis, and the direction of height along the z-axis. In 
this geometry, let the distance from the viewer to the stereoscopic image display screen be denoted by ds, 

35 the positions of the left and right eyes be represented by coordinates (-We, 0, 0) and (We, 0, 0), 
respectively, and the positions of reproduced spot images be designated Pr for the right eye and PI for the 
left eye. Then, the viewer will perceive the spot image lying at the point of intersection, P, of the line joining 
Pr and (We, 0, 0) and the line joining PI and (-We, 0, 0). Generally, in a stereoscopic image display 
apparatus having a flat screen such as shown in Figure 2(b), when dx = <», and when the position at which 

40 a point of infinity was captured (the center position of the captured image) is displayed at NL (-We, ds, 0) 
for the left image and at NR (We, ds, 0) for the right image, then the x direction of the imaging space, when 
reproduced, is free from distortion. Further, the y direction of the imaging space, when reproduced, is free 
from distortion when the following equation holds. 

45 [Mathematical!] ds = M»f 

where M is the ratio of the image on the imaging surface 7, 8 to the size of the image on the screen 12 
(provided that the center of the viewer is positioned on the y-axis). 

In this case, when viewing a reproduced image of a point of infinity, the visual axes of both eyes are 

so parallel to each other as shown by the dashed lines in Figure 2(b). Since the object is positioned 
somewhere between the point of infinity and the distance of 0, the visual axes of the viewer are either 
directed in parallel or turned inward, but are not turned outward. Since human eyes cannot be turned 
outward of the parallel lines at the same time, the above provides the condition that achieves the widest 
possible range for the reproduction of a stereoscopic image. 

55 Of the above distortion-free reproduction conditions, in cases where the condition of dx = °° in Figure 
2(a) is not satisfied, that is, in cases where dx takes a finite value, the resulting stereoscopic image always 
suffers spatial distortion. Therefore, the condition of dx = <» should be observed if at all possible, and only 
when the resulting stereoscopic image does not lie within the fusing range of the viewer's eyes, dx is varied 

5 
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so that the viewer can fuse the images into a single image. In this case, the value of dx. should be set as 
large as possible to minimize the distortion of the reproduced stereoscopic image. How dx is actually 
controlled will be described later. At the stereoscopic image reproduction side, images of a point of infinity 
are reproduced at NR and NL shown in Figure 2, for maximum reproduction range of the stereoscopic 
image. To achieve this, since the horizontal coordinate of the point of infinity on the imaging surface 7, 8 in 
Figure 2(a) is fx Wc / dx, AS is set so that the following equation holds, displaying the left and right images 
by shifting the right image by AS horizontally to the right and the left image by AS horizontally to the left. 



[Mathematical 2] 



w M-f-Wc AS 

dx 



Next, the binocular fusing characteristics of a human will be described. In the present invention, the 
parameter dx in the image pickup apparatus is controlled so that when a stereoscopic image is displayed 
by the stereoscopic display apparatus, the stereoscopic image can be displayed within the range in which 
the viewer can fuse the left and right images into a signal image. It is therefore necessary that the condition 
that enables the viewer to achieve binocular fusion be first determined in the geometry shown in Figure 2- 
(b). In Figure 2(b), suppose, for example, that the viewer is viewing the point P. In this case, the actual 
images are focused on the display screen 12. However, the visual axes of both eyes are directed to the 
point P. Crossing both eyes inward to view a near object is known as convergence eyemovement, and the 
point P is called the converging point The convergence eyemovement is controlled using the eye's focus 
information as well as the information concerning the projected position of the point P on the retina. 

In this case, since there is a conflict between the focus information and the retina information, strain is 
imposed on the viewers eyes. When the conflict remains within a limited range, the vieweV can tolerate the 
strain, but when the conflict becomes large, the viewer becomes unable to fuse the left and right images 
into a single image, and eventually perceives them as a double image. Even if the image does not appear 
double, it imposes strain on human's visual system, leading to fatigue. 

Allowable ranges for the fixating position of the eye (hereinafter called the accomodation point) and the 
position of the converging point (hereinafter called the convergence) based on the projected positions on 
the retinas were measured. The results are shown in Figure 3. The convergence plotted along the abscissa 
is represented by the distance between the viewer and the intersection of the visual axes of both eyes, and 
the accomodation point plotted along the ordinate is represented by the distance (measured in diopter, the 
reciprocal of the distance) between the viewer and the position on which the lenses of the eyeballs are 
focused. 

The straight line upward to the right at an angle of 45 • represents the condition where the convergence 
coincides with the accomodation point, the shaded area shows the allowable range within the depth of focus 
of the eye, and a indicates the fusional limits when an image is presented for 0.5 second. Within the region 
between the upper and lower fusional limits, the viewer can fuse the stereoscopic images and no double 
image will occur. In shooting an object, therefore, control is performed so that dx in Figure 2 will have the 
largest possible value while satisfying the condition that enables all stereoscopic images to be reproduced 
within the fusional range. 

First, dx is initially set at «> (sufficiently large value). The parallax calculator 15 shown in Figure 1 
calculates a parallax map (a three-dimensional map) from the left- and right-eye images output from the 
signal processors 14a and 14b. For calculation, there have been proposed a variety of methods, such as a 
correlation matching method that calculates the correlations of luminance patterns between left and right 
images, and an edge information matching method that involves matching the edge information between left 
and right images. The example hereinafter described employs the correlation matching method that 
calculates the correlations of luminance patterns. The operation of the parallax calculator 15 will be 
described in detail with reference to Figure 4. In Figure 4, consider left and right images each ofNxM 
size. In the left image, consider a block window of n x n pixels (3x3 pixels in the figure). The same image 
as shown in this block window is located in the right image by using a window of the same size. At this 
time, the displacement between the left and right blocks is represented by a vector (Ax, Ay), whose 
horizontal component Ax indicates the binocular parallax of the left and right images at the center 
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coordinate of the block window. By horizontally shifting the reference block window in the left image in 
sequence across the entire screen, and finding the corresponding block position (representing the binocular 
parallax) in the right image for each shifted block position, a parallax map (showing depthwise distance at 
each position on the screen) can be obtained for the entire screen. The displacement between the left and 
right images at coordinates (x, y), that is, the binocular parallax (Ax, Ay), can be expressed as 

[Mathematical 3] Ax = i , Ay=j for Min {Corr(ij)} 

where 

[Mathematical 4] 

nxn 

CorrCi, j)= Y, GL(Xk,yk)-GR(Xk-i,yk-j) 

k=1 

20 In the expression shown in Mathematical 3, r means taking the sum of the absolute values by varying the 
coordinates xk, yk within the block window of n x n. N is the number of pixels in the block window. In the 
binocular parallax Ax, Ay, the component that directly indicates the depthwise position is Ax. With the left 
image as the reference, if the value of the binocular parallax is positive, the right image is positioned to the 
right, and the left image to the left, with respect to the reference image, the object lying farther than the 

25 depthwise position at which the binocular parallax is zero; on the other hand, if the value of the binocular 
parallax is negative, this indicates that the object is positioned nearer than the depthwise position with zero 
binocular parallax. 

Based on the thus obtained parallax map (showing parallax Ax calculated at each coordinate position on 
the entire screen), the parallax processor 16 extracts the smallest parallax value (the parallax of the object 
30 reproduced at the position nearest to the viewer) from the entire screen, the extracted value being denoted 
as Axmin. Next, based on Axmin, the depth display position calculator 17 calculates the depth reproduction 
position Ypmin of the stereoscopic image on the stereoscopic image presentation diagram shown in Figure 
5, using the following equation. 



35 



40 



45 



[Mathematical 5] 



Y Pmi n = 2We ; dS 0 ' , A , 

(max) -M'AXmin-2-AS-f-2We 

(max) 



where Ypmin represents the y-coordinate of the distance from the origin of the display coordinate system to 
the position of the object in fixating. In Figure 5, the viewer's binocular fusing range shown in Figure 3 is 
superimposed on the display model of Figure 2(b). The shaded portion shows the binocular fusional region 
of the viewer. In the figure, F(ds)min corresponds to the lower curve in Figure 3. When screen viewing 

so distance ds = 71 cm, then F(ds)min = 29 cm, which means that the viewer can fuse images within the 
range from infinity to 29 cm in distance. 

As can be seen from Figure 3, when the viewing distance is 71 cm or greater, the viewer can fuse 
images within the range from infinity to F(ds)min. When the viewing distance is smaller than 71 cm, the 
binocular fusing range is also limited by the upper curve in Figure 3, thus limiting the binocular fusing range 

55 to within the distance from F(ds)max to F(ds)min. The following description deals with a case where the 
viewing distance is 71 cm or greater. The fusing range verifier 18 judges whether Ypmin lies on the 
boundary of the binocular fusing range. 
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The judgement is made by evaluating whether the following relation holds or not. 
[Mathematical 6] F(ds)min + AF*Ypmin2:F(ds)min-AF 

where AF is an arbitrary small value. When the relation shown in Mathematical 6 holds, the display position 
of the reproduced stereoscopic image is within the binocular fusing range; therefore, the fusing range 
verifier 18 supplies the current dx value as-is to the converging angle controller 19. When Ypmin satisfies 
the following relation 

[Mathematical 7] Ypmin>F(ds)min + AF 

the display position of reproduced Ypmin is completely inside the binocular fusing range. In this case the 
fusing range verifier 18 adds Adx to the current dx value and supplies the resulting new dx value to the 
converging angle controller 19. However, when $ to be described later (Figure 6) is close to 0, there occurs 
little change in the camera's optic axis angle e\ therefore, dx need not be updated. 
When Ypmin satisfies the following relation 

[Mathematical 8] Ypmin<F(ds)min-AF 

the image reproduced at Ypmin is beyond the viewer's binocular fusing range. In this case, the current dx is 
reduced by Adx, and the result is taken as the new dx. If the value obtained by subtracting Ax from the new 
dx is smaller than dxO (>0, to be described later), then dxO is taken as the new dx. Calculation of dxO will 
now be described. The fusing range verifier calculates dx such that the y-coordinate of Ypmin agrees with 
F(ds)min in Figure 5, and takes the result as dxO. 

The following equation is used to calculate dx, which is obtained by eliminating AS from equations in 
Mathematical 5 and 2, substituting F(ds)min for Yp, and solving the result for dx. 

[Mathematical 9] 

fjy_ 0 - 2F(ds)min-f -Wc- M 

2We ds + F(ds)min-M -Axmin 

Using the thus obtained dx, the converging angle controller 19 controls the rotation controllers 4a and 
4b to position the camera bodies 3a and 3b so that their optic axes are directed to the converging point 
shown in Figure 2(a). Figure 6(a) shows how the camera control angle $ is defined at this time. When e = 
0, the optic axes of the two cameras are parallel to each other. In the above example, dx is updated in 
increments of Adx, but in the case of the condition shown by Mathematical 8, the value of dx 0 given by 
Mathematical 9 may be directly substituted into dx. ~~ 

In this case, control is performed so that Ypmin, the point in the object nearest to the viewer, becomes 
equal to F(ds)min, a point near the limit of the fusing range. At this time, if Ypmin is within the binocular 
fusing range under the condition that makes the optic axes of the camera bodies 3a and 3b parallel to each 
other , then the optic axes of the two cameras are maintained parallel to each other. 

As described above, according to the present embodiment, the parallax of captured images is 
calculated, and the optic axes of the stereoscopic pickup cameras are controlled so that the smallest 
parallax value can be reproduced within the fusing range of the viewer's eyes. When the images thus 
captured are presented for viewing, the viewer can fuse the reproduced stereoscopic images over the 
widest possible range. 

Further, in the first embodiment, the amount of change in dx is large in the vicinity of 0 = 0, as shown 
in Figure 6(b); therefore, 6 may be made equal to 0 at dx_a>. | n this case, the initial value of dx is set as 
dx oo. 

Also, the curve shown in Figure 6(b) may be linearly approximated. Furthermore, in the conditional 
judgement expressions shown in Mathematical 6 to 8. an offset Yp_off may be added to Ypmin. and the 
judgement condition may be shifted along the direction of the y-axis so that the position of Ypmin comes 
within the fusing range (shaded region). 
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Figure 7 is a diagram showing the configuration of an image pickup section in a stereoscopic image 
pickup and display apparatus according to a second embodiment of the present invention. In the figure, the 
numerals 1 and 2 are lenses, 3a and 3b are camera bodies, 4a and 4b are rotation controllers, 5 is a 
camera spacing controller, 17 is a depth display position calculator for calculating the spatial position of an 

5 object nearest to the viewer, and 19 is a converging angle controller for controlling the rotation controllers 
4a and 4b so that the spatial position of the object nearest to the viewer comes within the fusing range of 
the viewer's eyes. The differences from the first embodiment are that the coordinates xO, yO of the nearest 
object are directly input to the depth display position calculator 17, and that a fusing range determiner is 
provided that calculates the value of dx to bring the object nearest to the viewer into the fusing range of the 

w viewer's eyes. 

The operation of the thus constructed stereoscopic image pickup section of the second embodiment 
will be described below. 

In the shooting condition shown in Figure 2(a), the coordinates of the point in an object nearest to the 
cameras are set as N <xO t yO, zO). Prior to shooting the object, the operator measures this distance and 
75 inputs xO f yO to the depth display position calculator 17. These values may be input using a calculator 
keyboard or any other device that can enter numeric data. Based on these input values, the depth display 
position calculator 17 calculates the y-coordinate of the display position of the point N in the stereoscopic 
image reproduction/display coordinate system shown in Figure 2(b). The calculation is made using the 
following equation. 

20 

[Mathematical 10] 



30 




35 

In Mathematical 10, AS is determined in accordance with Mathematical 2, as in the first embodiment of the 
invention. First, Ypmin is calculated with the condition of dx = «> (the condition that sets the optic axes of 
the two cameras parallel to each other). Next, using the thus calculated Ypmin, the fusing range determiner 
20 determines the converging point dx of the cameras, based on the following 

40 

relation. If 

[Mathematical 1 1 ] Ypmin> F(ds)min 

45 is satisfied, dx = «> (sufficiently large value) is taken; otherwise, the dx 0 given by Mathematical 9 is 

calculated, and the result is taken as the new dx. 

Based on the thus obtained dx t the converging angle controller 19 calculates the optic axis angle 0 of 
the two cameras. The angle d is the same as that shown in Figure 6. The rotation controllers 4a and 4b 
control the optic axes of the respective cameras so that the thus determined angle B can be provided. 

so As described above, according to the present embodiment, the position of the object nearest to the 
cameras is input, and the converging point dx is determined so that the object can be reproduced within the 
fusing range of the viewers eyes, in accordance with which the optic axes of the stereoscopic image pickup 
cameras are controlled. When the images thus captured are presented for viewing, the viewer can fuse the 
reproduced stereoscopic images over the widest possible range. 

55 In the second embodiment of the invention, the values xO, yO are input to the depth display position 
calculator 17 manually by the operator, but these values may be input automatically by using a distance- 
measuring device such as a ultrasonic sensor. Also, xO may be fixed to a predetermined value, and the 
value of yO only may be input. 
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Figure 8 is a diagram showing the configuration of an image pickup section in a stereoscopic image 
pickup and display apparatus according to a third embodiment of the present invention. The block 
construction is the same as that shown in Figure 1. In Figure 8, the numerals 1 and 2 are lenses. 3a and 3b 
are camera bodies, 4a and 4b are rotation controllers, 5 is a camera spacing controller, 14a and 14b are 
signal processors each for converting the camera signal into the luminance signal, 15 is a parallax calculator 
for calculating horizontal parallax from left and right images, 16 is a parallax processor for detecting the 
smallest parallax, 17 is a depth display position calculator for calculating the spatial position of an object 
farthest from the viewer, and 18 is a fusing range verifier for verifying whether the spatial position of the 
object farthest from the viewer is within the fusing range of the viewer's eyes. 

Further, 19 is a converging angle controller for controlling the rotation controllers 4a, 4b so that the 
spatial position of the object farthest from the viewer comes within the fusing range of the viewer's eyes 

The differences from the first embodiment are that the parallax processor 16 calculates the largest 
parallax value, that the depth display position calculator 17 calculates the spatial position of the object 
farthest from the viewer, and that the fusing range verifier 18 verifies whether the spatial position of the 
object farthest from the viewer is within the fusing range of the viewer's eyes. In the first embodiment 
control was performed based on the smallest parallax value and the nearest object. 

The operation of the thus constructed stereoscopic image pickup section of the third embodiment will 
be described below. 

In the first embodiment of the invention, the viewer's viewing distance was set greater than 71 cm the 
binocular fusing range was from » to F(ds)min. and ds was controlled so as to bring F(ds)min within the 
binocular fusing range of the viewer. When ds < 71 cm. the distance ranging from F(ds)max to F(ds)min is 
the binocular fus.ng range, as described earlier; if F(ds)min is to be brought within the fusing range, control 
can be performed in the same manner as described in the first embodiment, but in the third embodiment of 
the invention, control is performed in such a manner as to bring F(ds)max within the binocular fusing range 

First, dx is initially set at » (sufficiently large value). Next, the parallax calculator 15 calculates a 
parallax map by using the image signals obtained from the camera bodies 3a. 3b and signal processors 
14a. 14b, as in the first embodiment of the invention. The method of calculation, is exactly the same as that 
employed in the first embodiment. Based on the parallax map. the parallax processor calculates the largest 
parallax value Axmax (the parallax of the object farthest from the cameras). Next, the depfh display position 
calculator 17, using Mathematical 5. calculates the position at which the farthest object is to be reproduced 
for display. The .coordinate system used here is the same as the stereoscopic image display coordinate 
system shown in Rgure 2(b). In Figure 9. the viewer's binocular fusing range shown in Figure 3 is 
superimposed on the display model of Figure 2(b). The shaded portion represents the region where the 
viewer's binocular fusing can be attained. Here. F(ds)max is given by the upper curve in Figure 3. 

At this time, if the value of Ypmax and the value of the viewer's far point binocular fusing limit satisfy 
the relation 

[Mathematical 12] F(ds)max-AF£Ypmax£F(ds)max 

the display position of the reproduced stereoscopic image is within the binocular fusing range. In this case 
the fus.ng range verifier 18 supplies the current dx value as-is to the converging angle controller 19* 
Further, if Ypmax satisfies the relation 

[Mathematical 13] Ypmax<F(ds)max-AF 



the display position of reproduced Ypmax is completely inside the binocular fusing range. In this case the 
current dx is reduced by Adx. and the result is taken as the new dx. If the value obtained by subtracting AF 
from the new dx is smaller than dxl (>0, to be described hereinafter), then dx1 is taken as the new dx. 
Calculation of dx1 will now be described. The fusing range verifier 18 calculates dx such that the y- 
so coordinate of Ypmax agrees with F(ds)max in Rgure 5, and takes the result as dx1 . 

The following equation is used to calculate dx, which is obtained by eliminating AS from Mathematical 5 
and 2, making Yp- F(ds)min , and solving the result for dx, as in the first embodiment of the invention. 
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[Mathematical 14] 



dx_i = 2F(ds)max-f-Wc-M 

2We ds+F(ds)max*M- AXmax 



io Further, if, as shown in Figure 9, Ypmax satisfies the relation 
[Mathematical 15] Ypmax> F(ds)max + AF 

the image reproduced at Ypmax is beyond the viewer's binocular fusing range. In this case, the fusing 
75 range verifier 18 adds Adx to the current dx and supplies the result to the converging angle controller 19 as 

the new dx. However, when 6 to be described later (Figure 6) is close to 0, dx need not be updated since 

there occurs little change in the camera optic axis angle $ described later. 

Using the thus obtained dx, the converging angle controller 19 controls the rotation controllers 4a and 

4b to position the camera bodies 3a and 3b so that the camera optic axes are directed to the converging 
20 point shown in Figure 2(a). The camera control angle 6 here is defined in the same way as defined in the 

first embodiment of the invention with reference to Figure 6(a); that is, when 0=0, the optic axes of the 

two cameras are parallel to each other. In the above example, dx is updated in increments of Adx, but in 

the case of the condition shown by Mathematical 15, the value of dx 1 given by Mathematical 14 may be 

directly substituted into dx. 

25 In the above operation, control is performed so that Ypmax, the point in the object farthest from the 
viewer, becomes equal to F(ds)max, the far point limit of the fusing range. At this time, if Ypmax does not 
come within the binocular fusing range even under the condition that makes the optic axes of the camera 
bodies 3a and 3b parallel to each other, the optic axes of the two cameras are maintamed parallel to each 
other. 

30 As described above, according to the present embodiment, the parallax of captured images is 
calculated, and the optic axes of the stereoscopic image pickup cameras are controlled so that the largest 
parallax value can be reproduced within the fusing range of the viewer's eyes. When the images thus 
captured are presented for viewing, the viewer can fuse the reproduced stereoscopic images over the 
widest possible range. 

35 In the first to third embodiments of the invention, noise may be introduced into the output of the 
parallax calculator 15, depending on the condition of the object being shot; therefore, the output of the 
parallax calculator 15 may be passed through a time-domain low-pass filter to reduce the noise. Further, 
control of the two cameras need only be performed slowly; to prevent abrupt movements, Axmin output 
from the parallax processor 16, dx output from the fusing range verifier 18, or dx output from the fusing 

40 range determiner 20 may be low-pass filtered in the direction of time. 

Figure 16 is a diagram showing the configuration of a display section in a stereoscopic image pickup 
and display apparatus according to a fourth embodiment of the present invention. In Figure 16, the numeral 
29 is a parallax calculator, 30 is an fixating point calculator, 31 is a parallax controller, and 32 is an image 
display. The operation of the thus constructed stereoscopic image display of the present embodiment will 

45 be described below. 

First, the parallax calculator 29 calculates a parallax map (a three-dimensional map) from left- and right- 
eye images. The same calculation method as used in the first embodiment will be used here, such as the 
correlation matching method that calculates the correlations of luminance patterns between the left and right 
images. That is, in Figure 4, consider a block window of n x n pixels (3 x 3 pixels in the figure) in the left 

so image. Then, the same image as shown in the block window is located in the right image by using a 
window of the same size. In the displacement (Ax, Ay) between the left and right block positions, the 
horizontal displacement component Ax represents the binocular parallax of the left and right images at the 
center coordinate of the block window. 

By horizontally shifting the reference block window in the left image in sequence across the entire 

55 screen, and finding the corresponding block position (representing the binocular parallax) in the right image 
for each shifted block position, a parallax map (showing depthwise distance at each position on the screen) 
can be obtained for the entire screen. For calculations, the same equation as used in the first embodiment 
is used, that is 
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[Mathematical 3] Ax = i , Ay = j for Min {Corr(ij)} 
where 



w 



15 



20 



25 



[Mathematical 4] 

nxn 

CorrO, J)= 2 GL(Xk,yk)-GR(Xk-i,y k -j) 

M 

th* HnZ d °" ^ t J" S f 0btained P ara,lax ma P lowing parallax Ax calculated at each coordinate position on 
LrZ , SCreen i; th H 6 f,xatm 9 P° int calculator 3° calculates an average va.ue for the parallax over the e"ti>e 
Smoles ofTo' 9 ? "T 96 V3lUe AX3Ve WitH 9reater W6i9ht at the center ° f the ^reen. Figure 7 shows 

weight coefficient K2 is assigned tolhe center r^Z KtTo tn ^rS i ^S^SK^b." 

&^r^.^~JS SSKT may be vaL 

[Mathematical 16] 



30 



Axave 



= Z{KxAX(i,j) 
•J 




X-Y(Klx|-+K2xX)| 



where 



40 
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TSS^^^^^ ™ f?"* SCreen> and Where K = K2 < re 9 io " A >- and K = Kl (region 
a), wnen K1 - K2,a common average is taken over the entire screen 

horliSS r 6 ' 396 , V !' Ue AX3Ve ° f ,he bin ° Cular parallax - the P ar a»ax controller 31 controls the 
showst h"£ Tscal ^ 3 h ima9eS ^ m0V6S the ima ° es in hori20nta ' directions F^re 8 

TaZ ob^TTL ^rZ °' Vid6 ° Si9naK P ° in,S AL and AR indicate the sa ™ P°in?in the 

same objert in the left and r.ght .mages, respectively. The binocular parallax AX at the point A first 

ZTof ^'b?n " the „ ,i9Ure - AS Sh ° Wn in the fi9Ure ' the r] ° hi ima 9 e is shi «ed e ave age va ue 
Axave of the binocular parallax, ,n a direction that cancels the parallax (i.e. the ho-bo-tal -ead f-'< ~ tf-H 
-mage ,s shifted by Axave). In this manner, the image portion having a br 2ct£ pLa£ of' Axave is 

magesrHer he'ent^ ^ ima96 (rt,protod ,n ,h * Sa ™ ^onZ Tw and ^ 

Soino ^ a^ fe in f ,mag « ^T 6 ' h0ri20nta " y :aS previous, y described ' this is the same as 
ranging LA and LB ,n F.gure 15, and when the control amount of binocular parallax is small the chanae 
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of Th« h T I 3nd r ' 9ht ' ma9eS 00 the SCreen of the ima °e display 32 so that the maximum 
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12 



BNSDOCID:<EP_ 0641 132A1 I > 



EP 0 641 132 A1 



eyes are not forced to diverge. Further, the fixating point calculator 30 calculates the smallest parallax 
value, instead of an average value for the parallax over the entire screen. When the calculated value is 
negative, the parallax controller 31 controls the displacement between the left and right images on the 
screen of the image display 32 so that the minimum value of the displacement does not become smaller 
5 than a prescribed amount £; with this setting, the viewpoint of the viewer becomes very near, eliminating a 
large disparity between the eye's focus information from the stereoscopic image display screen and the 
converging angle of the visual axes, and the left and right images can be controlled so that the viewer can 
easily fuse the images. 

Furthermore, if the amount of displayed image displacement is changed from Axave to Axave - a (a is 

io an arbitrary value), the portion of the parallax represented by Axave can be set to a prescribed binocular 
parallax value a. In this case, a = 0 indicates a position on the surface of the image display screen; this 
position moves forwardly or rearwardly of the image display screen as the value of a is varied. In this case, 
control is performed using one average binocular parallax value for one picture; on the other hand, in the 
case of a moving image, an average binocular parallax value may be obtained for each picture (one frame 

75 in an NTSC image), and the image display section may be controlled using a signal that is low-pass filtered 
in the direction of time with the average parallax values taken as time series data. In the first embodiment, 
only the right image is shifted; alternatively, the right image may be shifted by half the control amount of 
displacement while the left image is shifted in the opposite direction by the same amount. Also, instead of 
taking an average binocular parallax, a median (a median processing filter) may be used. 

20 In the present embodiment, the displayed images are controlled only in horizontal directions using the 
displacement Axave caused by horizontal parallax. In addition to this, if the image read position in the 
vertical direction is controlled by using Ayave based on the binocular parallax Ay calculated by the parallax 
calculator 29, stereoscopic images with reduced vertical parallax can be presented with enhanced visibility. 
(Vertical displacement between the left and right images would greatly interfere with the binocular fusing 

25 ability of the viewer.) 

As described above, according to the present embodiment, the stereoscopic image indicated by the 
average binocular parallax value (which may indicate the depthwise position of the center of the parallax 
map) can be controlled so that it will be reproduced at the surface of the display screen or at other desired 
position, thus enabling the viewer always to perceive the stereoscopic depth of the object over the widest 
30 possible range. 

Figure 19 is a diagram showing the configuration of a display section in a stereoscopic image pickup 
and display apparatus according to a fifth embodiment of the present invention. In Figure 19, the numeral 
29 is a parallax calculator, 31 is a parallax controller, and 32 is an image display. These components are 
the same as those shown in Figure 16. The difference from the configuration shown in Figure 16 is that the 

35 configuration of Figure 19 has two additional functions: a line-of-sight sensor 33, a line-of-sight detector 34, 
and an fixating point evaluator 35 are provided, which together serve the function of measuring the line of 
vision (visual axis) that points to the position in a displayed image on which the viewer's eye is focused; 
and a control amount calculator 36 is provided to serve the function of controlling the amount of binocular 
parallax on the basis of the output of the parallax calculator and the information concerning the viewers' line 

40 of vision. 

The operation of the thus constructed stereoscopic image display section of the fifth embodiment will , 
be described below. First, the parallax calculator 29 calculates a parallax map (a three-dimensional map) 
from left- and right-eye images, as in the fourth embodiment of the invention. The same calculation method 
as 1 used in the fourth embodiment may be used for the above calculation; for example, the correlation 

45 matching method may be used which calculates the correlations of luminance patterns between the left and 
right images using the algorithm shown in Figure 4. Based on the thus obtained parallax map (showing 
parallax Ax calculated at each coordinate position on the entire screen), and on the position (fixating point) 
the viewer is viewing on the display screen of the image display 32, the control amount calculator 36 
calculates the amount of displacement in horizontal read timing between the left and right images input as 

so signals to the parallax controller 31. 

We will now describe a method of detecting the viewer's fixating point (the point on which the viewer's 
eye is focused). First, the direction of the visual axis of the viewer's eyeball is detected using the line-of- 
sight sensor 33 and line-of-sight detector 34. There are many detection methods, such as a limbus 
reflection method, a corneal reflection method, and a method involving monitoring the eye using a TV 

55 camera. Any of these methods may be used, but the following description deals specifically with the limbus 
reflection method. Figure 20 shows the operating principle of the line-of-sight sensor 33 in the limbus 
reflection method. Figure 20(a) is a front view of an eyeball, and (b) is a side view of the same. To detect a 
horizontal movement of the eyeball, the eyeball is illuminated with a weak infrared beam from an infrared 
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LED and the infrared beam reflected from the eyeball is measured with photodiodes. The photodiodes are 

? f f,r!° nS °" b ° th Sid6S ° f the iris as shown bv dashed lines - and the difference between the 
outputs of the photodiodes is calculated by an operational amplifier, to detect a horizontal movement of the 
eyeba . To detect a vertical movement of the eyeball, an infrared beam is directed to the lower part of the 
eyeball, as shown in Figure 20(b). and the reflected light is detected using a photodiode directed to the 
ower part of the ins. The thus obtained horizontal/vertical eyeball movement signals are converted by the 
.ne-of-s.ght detector 34 to a line-of-sight angle (line-of-sight signal). That is, the line of vision is measured in 
terms of horizontal angles, ax and ay. with respect to a predetermined reference point (the optic axis of the 
viewer s eyeball .s directed in the direction indicated by the angles). Next, based on the line-of-sight signal 
he f.xatmg point evaluator 35 calculates the position on the image display 32 on which the viewer's eye is 
focused More specifically, when the distance from the viewer to the display screen of the image display 32 

rJrlTf y . - eferenCe P ° int iS S6t 3t the Center of the dis P lav screen ' and a two-dimensional 

coordinate system havmg ,ts origin at the display screen center is defined, as shown in Figure 21 then 

using the l.ne-of-sight signal (ax. ay) the viewpoint F (x, y) defined in the two-dimensional screen coordinate 
system can be expressed as shown below. 

[Mathematical 17] x = Lxtan(ax) 
y = Lxtan(ay) 

Here, it is assumed that the viewer is positioned facing directly the center of the display screen, and that 
the display screen is not tilted with respect to the viewer. 

Using the thus obtained viewer's fixating point on the screen (the point in an image on which the 
viewer s fixating is focused), the fixating point evaluator 35 calculates the coordinates of the point on which 
the viewer s eye is rested. Figure 22(a) shows an example of time variation of the x component of the 
viewpoint F. When the viewer views points x1 to x5 in sequence as shown, the viewer's eyeballs move very 
quickly as the viewpoint changes from x1 to x2 to x3. and so on. Such eye movement is generally referred 
to as saccade. When the viewer blinks his eye, it is not possible to measure the viewpoint since rapid 
movements of the eyel.d cause abrupt changes in the waveform. The fixating point evaluator 35 removes 
such trans.ent states and detects the state (attention state) in which the viewer's eye is rested on any of the 
points xl to x5. More specifically, the fixating point evaluator 35 calculates the moving speed of the 
viewpoint, and determines that the viewer's eye is in the fixating state (marked * in Figure 22(b)) when the 
mov.ng speed is less than a prescribed value fi, as shown in Figure 22(b). In the above calculation, only the 
x component of the viewpoint is used, but the fixating state may be detected by calculating a two- 
dimensional speed using both the x and y components. The time variation of the thus extracted viewer's 
fixating point is shown in Figure 22(c). 

Next, as shown in Figure 22(d). positional data of the fixating point is filtered through a zero-order hold 
interpolating the time where no fixating point data is available. Thereafter, the time variation of the fixating 
point thus detected is smoothed by a low-lass filter. In this manner, the fixating point evaluator 35 detects 
he position in the delayed image on which the viewer's fixating is focused, and supplies the result as the 

SIS k°'h !' 9 T V Ue r COntr0 ' am ° Unt Ca ' CU,at0r 36 - ' n the above exam P' e ' t^ fixating point signal is 
created by detecting the line-of-sight direction of one eye. Alternatively, the line-of-sight directions of both 
eyes may be detected, and the fixating point signal may be calculated by using the average value of these 

Based on the thus obtained parallax map and fixating point signal, the control amount calculator 36 
determines the control amount by which the parallax controller 32 controls the horizontal read timing of the 
left and right images. This operation will be described with reference to Figure 23. F designates the viewer's 
fixating point indicated by an fixating point signal at a given time. With this point as the center, the binocular 
parallaxes Ax shown on the parallax map are averaged over the range of M x M pixels. The result is taken 
as the average binocular parallax Axave which was explained in the fourth embodiment of the invention 
Using this value, the parallax controller 31 controls the horizontal read timing of the left and right images 
and moves the images horizontally. The control method is the same as that explained in the fourth 
embodiment with reference to Figure 18. That is. the right image is shifted by the average binocular 
parallax Axave (the horizontal read timing of the image is shifted by Axave). 

In the fifth embodiment of the invention, the control amount calculator 36 takes the average of the 
binocular parallaxes within the square area having the fixating point as its center. Alternatively, their 
weighted average may be taken with greater weight at its center, as in the example shown in Figure 17 in 
the fourth embodiment. 
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Further, in the present embodiment, the line-of-sight signal is detected based only on the optic axis 
direction of the viewer. This method is effective when the viewer's head stays relatively still. In a freer 
viewing condition, a human's line of vision is given as a combined value of the direction in which the 
eyeball is pointed and the direction in which the head is moved. Methods of detecting the line of vision by 
5 combining the head and eyeball movements include a method of detecting the head of a viewer by a TV 
camera, and a method using a magnetic field generator in conjunction with a magnetic field detector 
mounted on the viewer's head, as disclosed in Japanese Patent Unexamined Publication No. 4-182226. Any 
of these methods may be employed. 

As described above, according to the present embodiment, since the binocular parallax at the position 
10 in an image that the viewer is currently viewing can be set at 0 (the image is reproduced at the position of 
the display screen) or at a desired value, the stereoscopic image is always formed around the image 
position intended by the viewer, enabling the viewer to perceive the stereoscopic depth of an object over 
the widest possible range. 

Figure 24 is a diagram showing the configuration of a display section in a stereoscopic image pickup 

75 and display apparatus according to a sixth embodiment of the present invention. Figure 24(a) shows a 
stereoscopic image recording section, and (b) shows a stereoscopic image display section. In Figure 24, 
the numeral 29 is a parallax calculator, 31 is a parallax controller, 32 is an image display, and 36 is a 
control amount calculator. These components are the same as those used in the fifth embodiment of the 
invention (Figure 19). The difference from the fifth embodiment is that a stereoscopic image pickup camera 

20 37, a recorder 38, an fixating request pointer 39, and a reproducer 40 are added. 

The operation of the thus constructed display section of the sixth embodiment will be described below. 
First, in Figure 24(a), the operator shoots an object using the stereoscopic image pickup camera 37. The 
stereoscopic image pickup camera 37 can be a conventional stereoscopic camera consisting of two 
ordinary video cameras mounted side by side on a universal head, or the stereoscopic image pickup 

25 section of the first embodiment of the invention. Captured images are recorded by the recorder (consisting 
of two conventional VTRs synchronized to each other). 

At this time, the operator inputs the position of the image on which the operator wishes the viewer to 
focus his fixating (an fixating request signal) by using the fixating request pointer 39. The fixating request 
signal is input using an ordinary pointing device, such as a mouse or tablet, and points to a given point in a 

30 captured image. This signal varies with time. The fixating request signal at a given time t is designated by 
Fix (xt, yt), where xt and yt are two-dimensional coordinate values designating the position in a captured 
image pointed to by the pointing device. This fixating request signal is recorded together with the captured 
image by using the recorder 38 (VTRs). More specifically, the fixating request signal is sampled at a field 
frequency of 60 Hz, and inserted in the video signal during each vertical retrace interval (vertical blanking 

35 period). 

In an alternative arrangement, the fixating request pointer 39 may be attached to an editing machine or 
a VTR so that the fixating request signal may be recorded when editing the recorded image. 

The thus recorded video signal and fixating request signal are processed in the stereoscopic image 
display section shown in Figure 24(b), and stereoscopic images are produced for display. The reproducer 

40 40 (VTRs) reproduces the video signal and fixating request signal. The remainder of the operation is the 
same as that described in the fifth embodiment of the invention (Figure 19), except that the fixating point 
signal is replaced by the fixating request signal output from the reproducer 40. That is, based on the 
parallax map calculated by the parallax calculator 29 and on the fixating request signal reproduced by the 
reproducer 40, the control amount calculator 36 determines the control amount by which the parallax 

45 controller 32 controls the horizontal read timing of the left and right images. As shown in Figure 23 in the 
fifth embodiment of the invention, the binocular parallaxes Ax shown on the parallax map are averaged over 
the M x M-pixel area centered around the image position on which the fixating request signal requests the 
viewer to focus his fixating. The result is taken as the average binocular parallax Axave which was explained 
in the fourth embodiment of the invention. Using this value, the parallax controller 31 controls the horizontal 

so read timing of the left and right images, and moves the images horizontally. The control method is the 
same as that explained with reference to Figure 18. That is, the right image is shifted by the average 
binocular parallax Axave (the horizontal read timing of the image is shifted by Axave). 

In Figure 24, instead of reproducing the fixating request signal by the reproducer 40, the output of the 
fixating request pointer 39 may be fed directly to the control amount calculator 36 so that the viewer may 

55 input the fixating request signal while viewing the reproduced image. 

The fourth to sixth embodiment have dealt only with binocular stereoscopic images, but it will be 
recognized that, in a multinocular stereoscopic image display apparatus also, the same processing as 
described above can be applied to two images to be projected into the viewer's left and right eyes. In this 
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case however, since a plurality of left- and right-eye image pairs are presented for viewing at the same 
time, the same processing needs to be applied to each of the images. 

As described above, according to the present embodiment, by controlling the readout position of the 
stereoscopic .mage using the fixating request signal, the visual program producer, the camera operator or 
the v.ewer can bring the desired portion of the image onto the surface of the display screen or onto other 
desired position; as a result, the stereoscopic image is always formed with a wide fusing range around the 
image position .ntended by the viewer, enabling the viewer to perceive the stereoscopic depth of an object 
over the widest possible range. 

Figure 25 is a diagram showing the configuration of a stereoscopic image pickup and display apparatus 
according to a seventh embodiment of the present invention. In the figure, the numerals 1 and 2 are lenses 
3a and 3b are camera bodies, 4a and 4b are rotation controllers, 5 is a camera spacing controller. 14a and 
14b are signal processors each for converting the camera signal into the luminance signal. 15 is a parallax 
calculator for calculating horizontal parallax from left and right images, 16 is a parallax processor for 
detecting the smallest parallax, 17 is a depth display position calculator for calculating the spatial position of 
an object nearest to the viewer. 18 is a fusing range verifier for verifying whether the spatial position of the 
object nearest to the viewer is within the fusing range of the viewer's eyes, and 19 is a converging anqle 
controller for controlling the rotation controllers 4a, 4b so that the spatial position of the object nearest to the 
viewer comes within the fusing range of the viewer's eyes. These components are the same as those of the 
image pickup section (Figure 1) in the stereoscopic image pickup and display apparatus of the present 
invention. Further, the numeral 30 is an fixating point calculator. 31 is a parallax controller, and 32 is an 
image display. These components are the same as those of the display section (Figure 16) in the 
stereoscopic image pickup and display apparatus of the fourth embodiment of the invention 

That is, in the seventh embodiment of the invention, both the image pickup section and the display 
section are controlled so that the presented stereoscopic image comes within the binocular fusing range of 
the viewer. The operation of the thus constructed stereoscopic image pickup and display apparatus of the 
seventh i embod.ment will be described below. First, in shooting an object, control is performed so that dx in 
Figure 2(a) will have the largest possible value while satisfying the condition that enables all stereoscopic 
images to be reproduced within the fusional range. This operation is exactly the same as that described in 
the first embod.ment of the invention. A brief description will be given here. To achieve the above condition 
dx is m.t.ally set to ~ (sufficiently large value). In Figure 25, the parallax calculator 15 calculates a parallax" 
map (a three-dimensional map showing parallax Ax calculated at each coordinate position on the entire 
screen) from the left- and right-eye images output from the signal processors 14a and 14b 

Based on this parallax map, the parallax processor 16 extracts the smallest parallax (the parallax of the 
object reproduced nearest to the viewer) from the entire screen, and takes the result as Axmin. Next, based 
on Axm.n. the depth display position calculator 17 calculates the depth display position Ypmin of the 
stereoscopic image in the stereoscopic image presentation diagram shown in Figure 5. as follows. 

[Mathematical 5] 



40 



Ypmin- 2We<ds 

(max) -M-AXmin-2-ASH-2We 
45 (max) 

If this Ypmin is within the binocular fusing range, the current dx value is taken as is. If 
so [Mathematical 7] Ypmin> F(ds)min + AF 
a small value Adx is added to dx. If 
[Mathematical 8] Ypmin<F(ds)min-AF 



55 



a small value Adx is subtracted from dx. These calculations are exactly the same as those described in the 
first embod.ment of the invention. Using the thus obtained dx value, the converging angle controller 19 
controls the rotation controllers 4a and 4b. In this manner, control is performed so that Ypmin. the point in 
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the object nearest to the viewer, becomes equal to F(ds)min ( a point near the limit of the fusing range. 

Using the output of the parallax calculator 15, the fixating point calculator 30 obtains the average 
parallax or the weighted average parallax Axave for the entire screen, in accordance with which the parallax 
controller 31 scrolls the left and right images in such a manner as to eliminate the displacement of Axave 
5 between the left and right images. This processing is exactly the same as that described in the fourth 
embodiment of the invention. 

In the above operation, the converging point of the image pickup section and the displayed image 
position are controlled so that the stereoscopic image displayed by the image display 32 comes within the 
binocular fusing range of the viewer. As a result, stereoscopic images with wide binocular fusing range can 
70 always be presented to the viewer, and eye strain during viewing can be reduced. 

In the above description of the seventh embodiment of the invention, the converging angle controller 19 
and parallax controller 31 are operated simultaneously. Alternatively, to ensure the stability of the entire 
system operation, these two controllers may be operated in turn, or one controller may be operated after 
the operation of the other controller has stabilized. 
75 As described, according to the present invention, binocular parallax between images is detected, based 
on which the optimum converging point of the cameras is calculated that enables the viewer to perceive the 
stereoscopic depth of an object with the widest possible range, and the cameras are controlled so that their 
optic axes are directed to the converging point; further, the displayed image is automatically controlled so 
that the widest possible binocular fusing range can be provided for the displayed image, thereby achieving 
20 the presentation of stereoscopic images that help to reduce eye strain. 

Claims 

1. A stereoscopic image pickup apparatus comprising: 

25 an image pickup device for capturing an object from a plurality of viewpoints; 

a converging angle moving mechanism for varying a converging angle of said image pickup device; 
a signal processor for extracting image data from said image pickup device; 

a parallax calculator for calculating a parallax of an image by using an output of said signal 
processor; 

30 a parallax processor for detecting the smallest value (a parallax of the nearest object) out of outputs 

from said parallax calculator; 

a depth display position calculator for calculating from an output of said parallax processor a 

depthwise position of an object to be reproduced at the nearest point when a captured image is 

displayed by a stereoscopic image display device; 
35 a fusing range verifier for verifying on the basis of an output of said depth display position 

calculator whether said depthwise position is within a fusional range of a viewer when the viewer views 

the reproduced image; and 

a converging angle controller for controlling said converging angle moving mechanism on the basis 

of an output of said fusing range verifier so that the object at the nearest point is positioned within the 
40 fusional range of the viewer. 

2. A stereoscopic image pickup apparatus according to claim 1 ,wherein 

the parallax processor detects the largest value (a parallax of the farthest object) out of outputs 

from said parallax calculator; 
45 the depth display position calculator calculates from an output of said parallax processor the 

depthwise position of an object to be reproduced at the farthest point when a captured image is 

displayed by a stereoscopic image display device; 

said fusing range verifier verifies whether said farthest point is within a fusional range of a viewer 

when the viewer views the reproduced image; and 
50 said converging angle controller controls said converging angle moving mechanism on the basis of 

an output of said fusing range verifier so that the object at the farthest point is positioned within the 

fusional range of the viewer. 

3. A stereoscopic image pickup apparatus comprising: 

55 an image pickup device for capturing an object from a plurality of viewpoints; 

a converging angle moving mechanism for varying the converging angle of said image pickup 
device; 

a depth display position calculator for calculating, by using positional data of the nearest of the 
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captured objects, a depthwise position of an object to be reproduced at the nearest point when a 
captured image is displayed by a stereoscopic image display device; 

a fusing range determiner for calculating from an output of said'depth display position calculator a 
condition that achieves a reproduction of an image of the object at said nearest point within a binocular 
fusing range of a viewer ; and 

a converging angle controller for controlling said converging angle moving mechanism on the basis 
of an output of the fusing range determiner. 

4. A stereoscopic image pickup apparatus according to claims 3, wherein: 

» said depth display position calculator calculates, by using positional data of the farthest of the 

captured objects, a depthwise position of an object to be reproduced at the farthest point when a 
captured image is displayed by a stereoscopic image display device; 

said fusing range determiner determines whether the farthest point is within a fusional range of a 
viewer when the viewer views the reproduced image; and 

the converging angle controller controls said converging angle moving mechanism on the basis of 
an output of the fusing range determiner so that said farthest point comes within the fusional range of 
the viewer viewing the reproduced image. 

5. A stereoscopic image display apparatus comprising: 

a parallax calculator for calculating a binocular parallax or three-dimensional position of an object 
by using images of the object captured from at least two different angles; 

an fixating point calculator for calculating a depthwise middle position of the object from an output 
of said parallax calculator; and 

a parallax controller for performing control so that a depthwise center position indicated by an 
output of said fixating point calculator is reproduced at a surface of a stereoscopic image display or at 
a designated depthwise distance from said surface. 

6. A stereoscopic image display apparatus according to claim 5, wherein 

said fixating point calculator calculates the depthwise position to be controlled" by said parallax 
controller, by taking an average or an weighted average with greater weight given to the center of the 
screen, over an entire screen or over a small area within an image in the output of said parallax 
calculator. K 

7. A stereoscopic image display apparatus according to claim 5, wherein 

said fixating point calculator calculates the depthwise position to be controlled by said parallax 
controller, by calculating the largest value or smallest value in an entire screen, or in a small area within 
an image, in the output of said parallax calculator. 

8. A stereoscopic image display apparatus according to claim 5, wherein 

said fixating point calculator includes a line-of-sight detector for detecting a line of vision of a 
viewer; 

an fixating point evaluator for detecting from an output of said line-of-sight detector a point in a 
displayed image on which the viewers fixating is focused; and 

a control amount calculator for calculating an average value for binocular parallax at a required 
display image position from an output of said parallax calculator and an output of said fixatinq point 
evaluator. y K 

9. A stereoscopic image display apparatus according to claim 8, wherein 

said fixating point evaluator extracts data only for a period in which a moving speed of the viewer's 
line of viS!on is slow, and holds and low-pass filters the data to detect the viewer's fixating point. 

10. A stereoscopic image display apparatus according to claim 8, wherein 

said control amount calculator calculates the depthwise position to be controlled by the parallax 
controller, by averaging the output of said parallax calculator over a prescribed range centering around 
the viewer's fixating point in the displayed image indicated by the output of said fixating point evaluator. 

11. A stereoscopic image display apparatus comprising: 

an fixating request pointer for specifying a portion of a captured image on which an operator 
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wishes a viewer to focus his fixating; 

a signal recorder for recording an output of said fixating request pointer and images of an object 
captured from at least two different angles; 

a reproducer for reproducing the recorded images and an fixating request signal; 
5 a parallax calculator for calculating a binocular parallax or three-dimensional position of the object; 

and 

a parallax controller for performing control on the basis of an output of said parallax calculator as 
well as the fixating request signal so that a image portion indicated by the fixating request signal is 
reproduced at a surface of a stereoscopic image display or at a designated depthwise distance from 
io said surface. 

12. A stereoscopic image display apparatus according to claim 11, wherein 

the output of said fixating request pointer is directly used as the fixating request signal instead of 
the fixating request signal reproduced by said reproducer. 

75 

13. A stereoscopic image pickup and display apparatus comprising: 

an image pickup device for capturing an object from a plurality of viewpoints; 
a converging angle moving mechanism for varying a converging angle of said image pickup device; 
a signal processor for extracting image data from said image pickup device; 
20 a parallax calculator for calculating a parallax of an image by using an output of said signal 

processor; 

a parallax processor for detecting the smallest value (a parallax of the nearest object) out of outputs 
from said parallax calculator; 

a depth display position calculator for calculating from an output of said parallax processor a 
25 depthwise position of an object to be reproduced at the nearest point when a captured image is 

displayed by a stereoscopic image display device; 

a fusing range verifier for verifying on the basis of an output of said depth display position 
calculator whether said depthwise position is within a fusional range of a viewer when the viewer views 
the reproduced image; 

30 a converging angle controller for controlling said converging angle moving mechanism on the basis 

of an output of said fusing range verifier so that the object at the nearest point is positioned within the 

fusional range of the viewer; 

an fixating point calculator for calculating a depthwise center position of the object on the basis of 

an output of said parallax calculator; and 
35 a parallax controller for performing control so that the depthwise center position indicated by an 

output of said fixating point calculator is reproduced at a surface of said stereoscopic image display 

device or at a designated depthwise distance from said surface. 

14. A stereoscopic image pickup and display apparatus comprising: 

40 an image pickup device for capturing an object from a plurality of viewpoints; 

a converging angle moving mechanism for varying the converging angle of said image pickup 
device; 

a signal processor for extracting image data from said image pickup device; 

a parallax calculator for calculating a parallax of an image by using an output of said signal 
45 processor; 

a parallax processor for detecting the smallest value (a parallax of the nearest object) out of outputs 
from said parallax calculator; 

a depth display position calculator for calculating from an output of said parallax processor a 
depthwise position of an object to be reproduced at the nearest point when a captured image is 
so displayed by a stereoscopic image display device; 

a fusing range verifier for verifying on the basis of an output of said depth display position 
calculator whether said depthwise position is within a fusional range of a viewer when the viewer views 
the reproduced image; 

a converging angle controller for controlling said converging angle moving mechanism on the basis 
55 of an output of said fusing range verifier so that the object at the nearest point is positioned within the 
fusional range of the viewer; 

an fixating request pointer for specifying a portion of a captured image on which an operator 
wishes the viewer to focus his fixating; 
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a signal recorder for recording an output of said fixating request pointer and images of an object 
captured from at least two different angles; 

a reproducer for reproducing the recorded images and an fixating request signal; 

a parallax calculator for calculating the binocular parallax or three-dimensional position of the 
object; and 

a parallax controller for performing control on the basis of an output of said parallax calculator as 
well as the fixating request signal so that an image portion indicated by the fixating request signal is 
reproduced at a surface of said stereoscopic image display device or at a designated depthwise 
distance from said surface. 
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